Note that the following discussion may refer to a number of publications and references. Discussion of such publications herein is given for more complete background of the scientific principles and is not to be construed as an admission that such publications are prior art for patentability determination purposes.
Automatic eye disease screening by computer implemented methods is being pursued as a means to provide safe and effective screening to the more than 26 million people with diabetes in the US who need at least one screen per year and are expected to double by the year 2050. Other parts of the world such as India, Brazil, Mexico, and China share this pressing need to care for their people. However, automatic methods rely on the quality of acquired digital images to be effective and, in turn, image quality relies on photographer's ability to acquire images. Rates of unusable images in screening settings have been shown to vary from 10% to 20% depending on photographers' skills. Embodiments of the present invention relate to methods to minimize image unusable rates, e.g. 5%, and enhance the photographer's skill. Embodiments of the present invention comprise computer implemented methods for real time visual image quality feedback that can be integrated into an image acquisition apparatus as part of an eye disease screening system delivered as part of primary care.
A great effort of the research community has been directed towards the automation of computer screening systems to detect eye disease in digital fundus images. However, these systems rely on images of adequate quality in order to produce accurate results. In a DR screening system, for example, an image is deemed as inadequate when it is difficult or impossible to make a reliable clinical judgment regarding the presence or absence of DR in the image. Studies show that the percentage of images that are inadequate for screening systems is about 10% of the mydriatic (pupil dilation) images. For single field non-mydriatic (no pupil dilation) images, the percentage of inadequate quality images has been reported at 20.8%. Major causes of inadequate image quality in retinal screening images include illumination crescents due to small pupil size, loss of contrast due to poor focus or movement of the eye or media opacity, and imaging of part of the eyelid and eyelash due to blinking, as well as insufficient illumination. Inadequate images reduce the utility of automatic screening systems because they have to be discarded from analysis, or cause preventable errors in said systems.
An automatic method to assess image quality is thus a needed pre-processing step prior to an automatic eye disease screening method. Such an image quality assessment method has two main tasks. The first task is to provide visual feedback to the user as to the quality of the image, the possible source of inadequate image quality, and steps that can be taken to correct for image defects such as retaking the images while the imaging subject is still present. The second task is to assign image quality-related labels to images before forwarding for further processing or expert evaluation. These two tasks are preferably performed in real time, while the patient is still being imaged, by a computing unit that executes computer implemented algorithms and is integrated with the image acquisition system, e.g. a retinal camera.
An automatic method to assess image quality can also perform the additional task of assigning disease-related labels to images prior to further processing or expert evaluation. For example, in DR screening, the presence of advanced stages of disease is correlated to length of time a person has had diabetes. Further, image quality is also negatively correlated to subject's age, and presence and level of retinal disease. Thus, low quality images from a person can be assigned disease-related labels such as “high DR risk”, or “refer”.
Desktop non-mydriatic retinal cameras comprise visual aids to help the person taking the images determine whether the camera is at the correct working distance from the patient's eye and whether the retina is in focus. However, only after an image is taken does the photographer can assess the quality of the image. Besides working distance and focus, image quality factors comprise compliance with desired field of view, i.e. whether the right part of the fundus is imaged, obscuration of parts of the fundus due to alignment and illumination, which can generate crescents and shadows, debris in the optical path, smears on the camera lens, and media clarity. Taking more images after visual inspection of image quality may solve some of the problems but at the expense of time since more images are needed, the comfort of the patient since more flashes of high intensity light are needed, and, paradoxically, detriment of image quality since the patient's pupil may not reach optimum natural dilation after just a three or four images are taken. Therefore, visual aids currently available in desktop non-mydriatic cameras are insufficient to ensure efficient imaging time and quality.
The likelihood of high percentages of unusable images prevent wide adoption of retinal screening at the primary care setting and limit the clinical utility of currently available systems. Embodiments of the present invention overcome these adoption barriers and helps ensure clinical utility, thus increasing the reach of retinal screening to patients at risk who live in areas where optometry and ophthalmology have little to no reach.
Today's commercial cameras have limited clinical utility at the primary care setting because they do not ensure that rates of unusable images will be low enough to justify the investment of $25,000 or more per camera. Even low-cost camera alternatives are difficult to justify when their efficiency depends heavily on photographer skills. The present invention renders current retinal cameras clinically viable in the primary care setting. Further economic gains can be realized when the present invention is integrated into a low-cost retinal camera through an embedded computing unit as described in one of the embodiments herein.
Obtaining the highest possible image quality is critically important when photographing a patient's retina in a clinic or collecting images of a subject for a study or clinical trial. Often the photographer taking the image will not appreciate the requisite criteria for image quality required by the end user, whether an ophthalmologist or a highly trained grader or reader of the retinal images. What may appear acceptable to the photographer may be deemed unacceptable or entirely unusable by the grader. In telemedicine, transmitting an unacceptable quality image may mean, at worse a missed diagnosis, or at best the need to retake the image or images at an inconvenience to a patient who will have to return to the clinic for re-imaging. In longitudinal studies where every image is critically important, losing an image for a given examination period may result in loss of that individual from the study or a reduction in the statistical power of the results.
High-quality images are a prerequisite for automatic screening systems, such as those that employ machine-coded mathematical algorithms to determine whether images include detectable signs of disease such as diabetic retinopathy. Retinal image quality is different from image quality in other medical imaging modalities and also from recreational photography, e.g. face portraits and landscapes. In retinal imaging, image quality is related to the level of confidence that a human reader has about his/her ability to make a clinical determination of the likelihood of the presence of eye disease. Retinal image quality issues can be divided into four general areas as listed below but other will be known to those skilled in the art:
1) physics-dependent issues comprise contrast, resolution, noise, artifacts, and distortion;
2) grader- and photographer-dependent issues comprise visual perception, training, and experience;
3) task-dependent issues comprise quantitative accuracy, sensitivity, specificity, and diagnostic accuracy; and
4) patient-depended issues comprise lens and media opacities, ocular aberrations, and retinal pigmentation.
Ultimately, an automatic image quality system must consider each of these issues and provide the photographer or user with descriptive labels as to why an image may be less than optimal.
While certain types of technical errors, e.g. poor alignment, pupil centering, blinks, can be corrected by re-acquiring the retinal image, others such as camera artifacts, e.g. glares, dust, scratches on optics, etc., cannot. Some patient effects, e.g. media opacities, pupil size, poor fixation, cannot be corrected to improve image quality, but immediate feedback to the photographer can identify the cause of the problem and suggest possible avenues for mitigating the problem. These technical errors must be identified in real-time by any image quality system preferably during the alignment process and before a retinal image is acquired. This process of assessing image quality during alignment helps prevent unnecessary flash exposures to the patient whose level of discomfort may increase with each flash and whose pupils may not return to a naturally dilated stage after a few flashes of the visible light used to acquire the retinal images. Currently, there are no retinal cameras that provide real time image quality assessment during alignment or after an image is acquired. Some desktop retinal cameras provide visual aids for alignment and focus but these are not sufficient to guide the photographer to ensure maximum image quality.
In multi-site study of 2771 patients where 304 (11%) of the images were found unreadable, approximately 25% were due to poor patient fixation, 25% due to poor focus, 25% due to small pupil size or media opacity. The remaining cause(s) for unreadable images was undeterminable. The proposed image quality system will detect poor quality images that are a result of these factors, and will offer to the photographer a “help” window to correct the problem.
Depending on the application, the quality of an image is deemed deficient when it becomes difficult or impossible to make a meaningful clinical judgment regarding the presence or absence of signs of pathology in the image, as listed in Table 1. In a computerized analysis of retinal images, developers must account for images of varying quality and their impact on diagnostic results in their algorithms. For clinical studies, low quality images must be examined immediately by a photographer, a grader, or an ophthalmologist and reacquired, if needed. The development and testing of an image quality verification system based on quantitative methods that characterize image features as perceived by human users are a focus of the present invention.
TABLE 1Disc-centered imageMacula-centered imageACCEPT-Nasal retinal vessels Field of view or macula ABLEacceptable focus for in soft focus but gradablelesion detectionModerate amount of Reasonable contrastshadowing/maculaReasonable contrastOptic disc with gradable featuresLightning artifacts Lightning artifacts less less than 10% of than 10% of thethe whole imagewhole imageConfidence in Confidence in lesion lesion identificationidentificationUNAC-Distinct media Field of view or macula CEPTABLEhaziness or poor vessels: cannot discern photographicdue to haze or shadowfocusMore than 25% of More than 25% of image image with artifacts with artifacts leading leading to to unreadability of areaunreadability of areaLow confidence in Low confidence in lesion identificationlesion identification
Automatic image quality assessment has been the topic of intense study by a number of researchers in other fields of medicine as well as the general topic of image quality. Reference image-based methods (i.e. a quality comparison with the optimal image of each retina is made using various quantitative measures) are disadvantageous because a limited number of the good quality images might not be a “good” reference for the natural large variance encountered in retinal images acquired from screening. Other methods include vessel detection in the whole image for image quality assessment, measurement of vessel density in a region around the macula for macula-centered images, and splitting the field of view into sub-regions with different sizes of elliptical rings and different angles, wherein the local vessel density and 5-bin color histogram features were used to determine image quality. Some methods involve segmentation of retinal vessels and other anatomical structures, then applying additional criteria, such as detecting small vessels around the fovea, to determine image quality. The practical utility of these techniques is very low because they are computationally intensive (not suited for real-time feedback to a photographer) and have an inherent uncertainty regarding a valid or successful segmentation step (such as finding small vessels to determine the quality of a retinal image).
Other methods include an image management method that assigns descriptive labels to images according to image quality factors in order to indicate to a user possible actions such as “delete”, “keep”, “keep and enhance”. The utility of these methods is limited to “recreational” photography where the image quality factors include image brightness, contrast, and focus. In medical imaging screening methods, such as the subject matter of the present invention, said image quality factors are necessary but not sufficient to provide an indication of the adequacy of an image. These recreational photography methods fail to consider other factors involved in assessing image quality of medical images in general and retinal images in particular. In the case of retinal images there are three quality factors that are important and unique to this imaging modality.
The first one relates to the image adequacy with respect to a specific imaging protocol, for example, an imaging protocol that requires images of certain, pre-defined areas of the fundus, e.g. the macula or the optic disc. A retinal image can have adequate levels of brightness, contrast, or focus but if one of these areas is not present, the image is unusable for practical purposes. An image quality screening method thus has to include factors that include adequacy to imaging protocols and give the user the option of retaking an image.
The second factor is the assessment of artifacts that obscure parts of the images required for adequate screening. These factors are described herein as part of the present invention but briefly, they include artifacts due to human factors such as small pupils, eye blinks, and media opacities among others “physiologic image quality artifacts”, that will be obvious to those skilled in the art. The assessment of physiologic image quality artifacts is unique to retinal imaging and its implementation as computer-implemented methods is unique to embodiments of the present invention.
The third factor relates to assessment of “case completion”. In retinal imaging, a case is considered complete when a pre-determined minimum set of adequate images is collected. This minimum set relates to a specific imaging protocol designed to capture certain areas of the eye fundus for screening. An image quality screening method for retinal screening thus requires methods that determine when a case is complete. This image quality factor is unique to retinal screening and its implementation as machine-implemented methods is unique to the present invention. The combination of these three image quality factors is not only unique to retinal image screening but necessary for its utility in practice. Further, said combination of image quality factors has not been reported in the literature or applied in practice. As it will become clear from the description of methods herein, the present invention includes the combination of methods that implement the three image quality factors needed to make automatic retinal screening useful and practical.
Other methods do not require any segmentation of the image, such as integrating individual characteristics to give a better evaluation of image quality than the individual parts, primarily in other fields not directly related to medical applications. For example, spatial features based on wavelets have been used to show that image characteristics such as spatial frequency, noise, sharpness, brightness, contrast, and modulation transfer function could be encoded by the wavelets. These characteristics, in turn, relate to image quality. However, this technique is computationally intensive and not suitable for real-time applications. Another method uses clustering of multiscale Gaussian derivative filterbank responses to obtain a compact representation of image structures. Image Structure Clustering (ISC) provides a compact representation of the structures found in an image, and determines the most important set of structures present in a set of normal quality images, based on a clustering of the response vectors generated by a filter bank. Clustering of filter bank responses has been used for different applications in image processing. The system is based on the assumption that an image of sufficient quality will contain a definable distribution of image structures. A cluster filter bank response vector is used to represent the image structures found within an image. Using this compact representation together with raw histograms of the Red, Green, and Blue (RGB) color planes, a statistical classifier can be trained to distinguish normal from low quality images. Though effective in evaluating image quality (0.997 area under the area under the receiver operating curve, AUC), this approach was found to require up to 30 seconds to perform the necessary calculations, thus exceeding the practical limits for clinical, real-time applications.
Others methods use global image intensity histogram analysis and combinations of local and global histogram analysis to automatically assess image quality. The difference between a test image histogram and a model histogram from good quality images is used to give an image quality metric. Some studies perform an analysis of the vasculature in a circular area around the macula. The presence of small vessels in this area is used as an indicator of image quality. Some methods use computational models for evaluating the acceptability of retinal images based on several criteria related to image quality. Several steps are required, including segmenting retinal vessels. To evaluate the clarity of an image, retinal vessels in the vicinity of the fovea are counted and measured. The clear presence of these vessels indicates a high-quality image. Other factors that affect image quality are individually addressed and quantitative criteria are set for each. This technique is computationally burdensome and must integrate explicitly all possible factors, each treated independently with a different algorithm. The method requires a segmentation of the vasculature and other major anatomical structures to find the region of interest around the fovea. Detecting the failure of the segmentation in case of low image quality is not trivial and limits the robustness of the approach and its clinical utility.